A Portable InfiniBand Module for MPICH2/Nemesis: Design and Evaluation

With the emergence of multi-core-based processors, it is becoming significantly important to optimize both intra-node and inter-node communication in an MPI stack. MPICH2 group has recently introduced a new Nemesis-based MPI stack which provides highly optimized design for intra-node communication. It also provides modular design for different inter-node networks. Currently, the MPICH2/Nemesis stack has support for TCP/IP and Myrinet only. The TCP/IP interface allows this stack to run on the emerging InfiniBand network with IPoIB support. However, this approach does not deliver good performance and can not exploit the novel mechanisms and features provided by InfiniBand. In this paper, we take on the challenge of designing a portable InfiniBand network module (IB-netmod) for Nemesis. The IB-netmod is designed over the Verbs-level interface of InfiniBand and can take advantage of all features and mechanisms of InfiniBand. A complete design of the IB-netmod with the associated challenges are presented. A comprehensive performance evaluation (micro-benchmarks, collectives and applications) of the new Nemesis-IB design is carried out against the Nemesis TCP/IP (with IPoIB support on InfiniBand) and the native IB support of the MVAPICH2 stack. The new IB-netmod is able to deliver comparable performance to that of the native IB support of MVAPICH2. Compared to the MPICH2/IPoIB support for InfiniBand, the new design is able to deliver significant performance benefits. For NAMD application with 256 cores, the new IB-netmod is able to deliver 4% improvement compared to the latest MVAPICH2 release. To the best of our knowledge, this is the first IB-netmod design for the MPICH2/Nemesis framework. The next release of MVAPICH2 will be having this new IB-netmod support.